DOCUMENT RESUME 



ED 356 278 



TM 019 733 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



ABSTRACT 



Boser, Judith A.; Clark, Sheldon B. 

Response Rates in Mail Surveys: A Review of the 

Reviews • 

Apr 93 

21p,; Paper presented at the Annual Meeting of the 
American Educational Research Association (Atlanta, 
GA, April 11-16, 1993). 
Information Analyses (070) — Reports - 
Research/Techni cal (1A3) — Speeches /Conference 
Papers (150) 

MFOl/PCOl Plus Postage. 

Databases; ''^'Literature Reviews; '"'Mail Surveys; Meta 
Analysis; '^'Research Methodology; Research Reports; 
^Response Rates (Questionnaires) ; Responses; 
Scholarly Journals; Search Strategies 



As a predominant research method, the mail survey has 
been the subject of a considerable body of research. Several major 
reviews of research have been published that attempt to synthesize 
findings of empirical studies and identify techniques that stimulate 
responses. The review articles are examined, identifying procedures 
and potential publication sources. Reviews are limited to those that 
are comprehensive in nature, and that focus on identifying techniques 
that facilitate response rates in mail surveys. Through searches of 
computerized databases, 9 reviews were located, based on 329 source 
studies. There is considerable variation among the nine articles. It 
is evident that the choice of kejwords or descriptors and differences 
in the databases have influenced the outcomes of the reviewers* 
search efforts. There is great disparity in the ways in which the 
procedures used in the integrated review articles are documented. The 
next step will be to compare the mail survey response rate variables 
that were studied. Four tables present information about the reviews. 
Appendix A lists the journal of publication for the source articles 
across time, and Appendix B lists the source article publication 
sources. (SLD) 



Reproductions supplied by EDRS are the best that can be made ''^ 
from the original document. 



U.S. 0£PAfrrMENT or ECHJCATION 
Office of Educ«lionai Research and Improvemeni 

EDUCATIONAL RESOURCES INFORMATIOM 
CENTER (ERIC) 



(O^his docufn«nl has been reproduced as 
received Irom the person or ofganl^a^lon 
originalmg it 
ft Minor changes have been made >o improve 
reproduction quality 



Points of view Of op«nions staled m this docu- 
ment do not necessarily represent ollicial 
OERI position or policy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIO.ML RESOURCES 
INFORMATION CENTER (ERIC)." 



Response Rates in Mail Surveys: 
A Review of the Reviews 



by 

Judith A. Boser 

Bureau of Educational Research and Service 
212 Claxton Education Building 

University of Tennessee 
Knoxville, Tennessee 37996-3400 

and 

Sheldon B. Qark 

Sdence/Engineering Education Division 
Oak Ridge Institute for Science and Education 
P.O. Box 117 
Oak Ridge, Tennessee 37831-0117 



Paper Presented at the Annual Meeting of 
the American Educational Research Association 
Atlanta, Georgia 
April 12-16, 1993 



2 lEST COPY AVAILABLE 



Response Rates in Mail Surveys: 
A Review of the Reviews 

According to Dillman (1991), the predominant sample survey n\ethod used in this country is the 

mail survey. As such, it has been the subject of a considerable body of research. Most studies of mail survey 

methodology focus on specific design elements, but several major reviews of research that attempt to 

sjmthesize the findings of the independent empirical studies and to identify the techniques that reliably 

stimulate responses in mail surveys have been published. In addition, other less comprehensive reviews 

focusing on specific survey design variables have been published. The methods used by the researchers in 

these reviews have varied, and not all of the more recent studies have utilized quantitative analytical 

procedures. The relative merit of the findings are somewhat dependent on the way in which the reviews 

were conducted and reported, of course, but both types of reviews can provide valuable information for other 

researchers in this field regarding mail survey procedures and potential publication avenues for research on 

mail surveys. 

Reviews of research are expected to preface new research studies but are also sometimes conducted 
on a more extensive scale as independent endeavors. Much less attention has been placed on methods for 
conducting and reporting reviews of previous research studies than on new or primary research studies. 
Jackson (1980) focused on integrative reviews of research, documenting and evaluating various methods by 
which they had been accomplished. The use of meta-analysis provided a major change in comparing the 
results of multiple studies. Cooper (1989) noted that the reviewing of previous research studies had been 
facilitated by the recent availability of computerized literature searches and the development of 
quantitative procedures for analysis. 

The purpose of the present study was to examine the review articles themselves, identifying 
procedures and potential publication sources for use by those interested in the study of mail survey methods. 
For purposes of this study, review articles are limited to those that are comprehensive in nature and focus 
on identifying techniques that facilitate response rates in mail surveys. In the present context, 
''comprehensive'' is defined as including several survey design variables that have possible impacts on 
survey response rates, and "source studies" are the citations (published or unpublished, including books) 
referenced by the review author(s) that were included in their quantitative analyses or that contributed to 
the formulation of their conclusions in qualitative reviews. 

Method 

Procedures for Identifying Review Articles 

The first step in identifying review articles was to search computerized CD-ROM databases 
containing abstracts in the fields of psychology, sociology, business and marketing, and education. The 
databases used were PSYCLit (1974-1992), Sociofile (1974-1992), ABI/Inform (1987-1992), and ERIC (1966- 
1992). Yu and Cooper (1983) had used "response rate" and "survey" in searching for articles for their 
review. Yammarino, Skinner, and Childers (1991) used "response rates" and "mail surveys" as key words. 
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An initial list of abstracts from each database was compiled by using the key words "response rate" or 
"response rates" and "mail survey" or "mail surveys " The ABI/Inform database searched only for the 
singular forms, amending the plural to the singular for both "rates" and "surveys." Jackson (1980), in 
searching for review articles, had looked for articles under the headings of "literature reviews" and 
"research reviews." Further defining the scope of the present search by adding the term "research review" 
or 'literature review" was too limiting, however, sometimes producing no citations, so the more inclusive 
initial lists were used. Reference lists in the review articles identified through the computer searches were 
examined for other review articles not included in the computer databases. 

Four acceptable review articles were found in PSYCLit among the 109 articles identified as 
containing "response rate" or "response rates" and "mail survey" or "mail surveys." Four review articles 
were among the 74 abstracts listed from ABI/Inform, and two from the 66 drawn from Sociofile. There were 
no review articles that met the criteria for this study among the 137 abstracts produced by the ERIC 
searches. Some of the acceptable review articles appeared in more than one database. One review article 
was indexed in three of the databases, two were found in both PSYCLit and ABI/Inform. When duplication 
was eliminated, there were six review articles that met the criteria for inclusion in the present study. 
Examination of the reference lists from those articles produced three additional reviews for the total of 
nine that serve as the basis for this study (see References: Reviews Included in the Study). 
Selection of Reviews 

Some of the review articles initially identified were not included in this study because they 
focused solely on a specific aspect of survey design, such as monetary incentives (Armstrong, 1975), cover 
letter personalization (Worthen & Valcarce, 1985), or type of postage used (Armstrong & Lusk, 1987). One 
review (Yu & Cooper, 1983) that was not included was not limited to mail surveys, and it did not 
distinguish between mail surveys and other types of surveys in the presentation of many of the findings. 
Berdie, Anderson, and Niebuhr (1986) also went beyond mail surveys. Houston and Ford's (1976) review 
examined response speed and quality, rather than response rate, and was excluded from the present study. 
Reviews that did not contain or nuke available a list of the source articles, such as Goyder (1982), were 
also not included. Dillman (1991) cited specific studies to illustrate the points he made regarding specific 
mail sur\'ey techniques as part of a comprehensive system for conducting a survey. 
Variables 

For each review article, the following information was sought: date and journal in which the 
review article appeared, type of analysis used, criteria for inclusion of source articles, number of source 
articles, years and journals in which source articles were published, method by which source studies were 
identified. When discrepancies appeared between reference lists in the listings of source articles, the 
original articles were obtained to determine the correct reference listing. 
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Source Studies 

Some review articles (Bruvold & Comer, 1988; Fox, Crask, & Kim, 1988; Heberlein & Baamgartner, 
1973; Yammarino, Skinner, & Childers, 1991) provided reference lists of the articles used in the 
quantitative analysis (source studies) and separate lists of references providing background or introductory 
material. This clearly identified the source studies, although the titles of some of the source studies did 
not clearly delineate them as relevant for mail surveys. In Conant, Smart, and Walker (1990), the review 
of articles to detemnine response facilitation techniques was only the first stage in a more involved study. 
The relevant citations were included in a single table, making them easy to discern. 

For the other four reviews, the two researchers independently read the articles and determined 
which sources were background or general references and which should be included as source studies. They 
then compared their lists and resolved discrepancies through discussion and clarification of the criteria for 
inclusion. It was decided that references that were cited as leading to the conclusions of the review author 
should be included, even though they were unpublished works, books, or articles that appeared to be 
inappropriate because their titles indicated a focus on surveys other than those conducted through the 
mail. 

The following information for each source study was entered into a Fox Base/MAC database: 
author(s), year of publication, journal in which it was published, and which review articles had used the 
study. Citation as a reference in each of the nine review articles was coded as a dichotomous variable. 
Because of the small number of review articles and the varying procedure^i used in them, frequency 
distributions were used in analyzing the data in addition to descriptions of the reviewers' procedures. 

Results 

Publication of Review Articles 

Three of the nine reviews were published in Public Opinion Quarterly, three in marketing research 
journals (Joumal of the Market Research Society and Journal of Marketing Research), and one each in the 
Journal of Business Research, Joumal of Management, and the American Sociological Review (see Table 1). 
None of the reviews appeared in either education or psychological journals. 
Procedures Used in Review Articles 

Four of the reviews utilized quantitative analyses, while the other five could best be described as 
qualitative. In general, the quantitative review articles contained better documentation of the methods 
used than the qualitative reviews. For example, none of the five qualitative reviews contained 
information about how the source articles were identified or the criteria for inclusion. The four 
quantitative reviews, in contrast, were detailed in describing methods for locating the articles, selection 
criteria, and analysis. 
Locating Source Studies 

Heberlein and Baumgartner (1978), the earliest quantitative review, did not use computerized 
searches, as did authors of the latter quantitative reviews. Computerized search services were relatively 
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new at that time. Heberlein and Bauntgartner published their review in a sociology journal. The Sociofile 
indexing system in use today (which indexes journals in the field of sociology) only dates back to 1974, as 
does PSYCLit. Social SciSearch began in 1972. The primary source of articles for the Heberlein and 
Baumgartner review was an annotated bibliography by Potter, Sharp, Hendee, and Clark (1972) containing 
193 citations, supplemented by manual reviews of journals and citations in the published articles that were 
located. The review authors did not, however, identify the journals which were searched manually. 

Table 1 

Review Pub* No, Source 

Article Year Journal Articles Analysis 



Yammarino, Skinner, 



& Childers 


1991 


Public Opinion Quarterly 


115 


meta-analysis 


Conant, Snnart, & Walker 


1990 


Journal of the Market Research Soc. 


52 


qualitative 


Fox, Crask, & Kim 


1988 


Public Opinion (Quarterly 


40 


meta-analysis 


Bruvold & Comer 


1988 


Journal of Business Research 


212 


weighted-least-squares 
logit regression 


Harvey 


1987 


Journal of the Market Research Soc. 


129 


qualitative 


Duncan 


1979 


Journal of Management 


60 


qualitative 


Heberlein & Baumgartner 


1978 


American Sociological Review 


95 


quantitative 


Kanuk & Berenson 


1975 


Journal of Marketing Research 


69 


qualitative 


Linsky 


1975 


Public Opinion Quarterly 


57 


qualitative 



Bruvold and Comer (1988), began with the list of studies used by Heberlein and Baumgartner. This 
was supplemented by manual searches of five journals (Journal of Marketing Research, Journal of Marketing, 
Public Opinion Quarterly, Journal of Business Research, and Journal of Applied Psychology) for the years 
1964-1980. The reviewers also conducted computer searches on six databases: Social SciSearch, 
PsychlNFO, ABI/Inform, Sociological Abstracts, U.S. Political Science Documents and ERIC. For key 
words, they used "response rate" and "mail survey or mail questionnaire." They included only published 
journal articles and one reference published in a conference proceedings. 

Fox, Crask, and Kim (1988) conducted a manual search of three of the same journals as Bruvold and 
Comer (Journal of Marketing Research, Public Opinion Quarterly, and Journal of Applied Psychology) for 
the previous 25 years or, in the case of the Journal of Marketing Research, since 1964 when it began 
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publication. They also did a computerized search using PsychlNFO and examining the reference lists from 
relevant articles. The keywords used in the computer search were not identified. The reviewers used only 
published articles that were experimental studies. Response rate variables that were coded subjectively, 
such as "salience" that had been included in the Heberlein and Baumgaitner (1978) study, were not 
considered. 

Yammarino, Skinner, and Childers (1991) began by examining reference lists of previously 
published review articles, including Kanuk and Berenson (1975), Linsky (1975), Heberlein and Baumgartner 
(1978), Yu and Cooper (1983), Harvey (1987), and Fox, Crask, and Kim (1983). They then conducted 
computer searches of the following databases: ABI/Inform, ERIC, Social SciSearch, Sociological Abstracts, 
and PsycINFO. The keywords of "mail surveys,'' "response rates," were used and also combined with the 
names and synonyms for the 17 respoi\S€ rate variables targeted for study. To identify recent studies (1978 
and later), a manual search of each of the following ten journals was performed: American Sociological 
Review, Journal of the Academy of Marketing Science, Journal of Advertising Research, Journal of the 
American Statistical Association, Journal of Applied Psychology, Journal of Marketing, Journal of 
Marketing Research, Journal of the Market Research Society, Psychological Reports, and Public Opinion 
Quarterly. To be included, the source studies had to have a manipulated factor and had to include response 
rates for the various conditions. 
Selection Criteria Differences 

Six of the reviews limited their source citations to published articles (Duncan, 1979; Fox, Crask, & 
Kim, 1988) and a published book (Kanuk & Berenson, 1975) or proceedings (Bruvold k Comer, 1988; Conant, 
Smart, & Walker, 1990; Heberlein & Baumgartner, 1978). The other three reviews (Linsky, 1975; Harvey, 
1987; Yammarino, Skinner, & Childers, 1991) included one or more unpublished papers and, in the 
Yammarino, Skinner, and Childers review, five unpublished studies cited in a Public Opinion (Quarterly 
review by Armstrong and Lusk (1987). 

Each of the eight books was cited in only a single review. With one exception, the unpublished 
papers were also unique to a single review, as were the five unpublished studies cited by Armstrong and 
Lusk (1987). There was only one instance of duplication among the studies cited from meeting proceedings. 
Soiuxre Studies 

A total of 329 source studies (as defined earlier) was used in the nine review articles. Some source 
articles were sometimes referenced incorrectly, initially causing them to be considered additional articles. 
Differences (errors) occurred in publication year, journal pages and/or volume number, journal title, author, 
spelling of the author's name, and even title of the article. 
Scope of Soiurce Study Publication Dates 

The reviews appeared from 1975 through 1991, with no reviews appearing between the 1979 Duncan 
review and the 1987 review by Harvey. The number of source stiidies used in the reviews ranged from 40 to 
212. While all review autiiors had access to articles appearing in the early years, three reviews were 



ERLC 



5 

7 



deliberately limited in scope to specific time periods: Duncan, 1961-1978; Fox, Crask, and Kim, 1961-1984; 
and Conant, Smart, and Walker, 1974-1988 (see Table 2). 



Table 2 

Publication Years of Source Studies 



Review 
Article 



Pub, 1921 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 

Year nd* - 39 - 44 - 49 - 54 - 59 - 64 - 69 - 74 - 79 - 84 - 89 Total 



Range 



Yanwiarino, 1991 
Skinner, & 
Childers 




1 


3 


1 


6 


4 


7 


12 


25 


33 


15 


18 


115 


1937-86 


Conant.. 
Smart, & 
vv aiKer 


1990 


















1 


27 


18 


6 


52 


1974-88 


Fox, Crask, 
& Kim 


1988 














3 


6 


8 


15 


8 




40 


1961-84 


Bruvold 
& Comer 


1988 




c 
o 


Q 




1 J. 


1*4 




23 


42 


60 


24 


1 


212 


1921-85 


Harvey 


1987 


1 


3 


6 


7 


14 


10 


7 


18 


15 


20 


23 


5 


129 


1937-87 


Duncan 


1979 














1 


9 


18 


32 






60 


1961-78 


Heberlein 
& Buum- 
gartner 


1978 




6 


8 


6 


13 


13 


13 


18 


16 


2 






95 


1921-75 


Kanuk 
& Berenson 


1975 




1 


5 


6 


8 


8 


16 


12 


11 


2 






69 


1939-75 


Linsky 


1975 


1 


1 


5 


2 


14 


10 


6 


13 


5 








57 


1939-74 


Total in 
Database 




1 


9 


10 


12 


21 


21 


25 


32 


49 


83 


49 


17 


329 


1921-88 



^Undated 

Of the 329 source studies, 136 (41.3 percent) were cited in only one review. Each of the nine reviews 
contained at least one source study not found in any other review (see Table 3). Three source studies were 
cited in eight reviews, the maximum possible given the scope in publicatton years of the nine reviews. Six 
studies were dted in seven reviews, 22 in six reviews, 18 in five reviews, 37 in four reviews, 43 in three 
reviews, and 64 in two reviews. 
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Tables 

Incidence of Unique Source Studies 



Review Authors 


Publication Year 


Total Citations 


Unique Citations 




1QQ1 


115 


27 




v^wiidiii/ JiiUAiXf ciiiu VValKcr 


ioon 
lyyu 


52 


11 


21.1% 






W 


1 


2.5% 


Bruvold and Comer 


1988 


212 


38 


17.9% 


Harvey 


1987 


129 


29 


22.5% 


Duncan 


1979 


60 


8 


13.37o 


Heberlein and Baumgartiier 


1978 


95 


1 


1.1% 


Kanuk and Berenson 


1975 


69 


15 


21.7% 


Linsky 


1975 


57 


6 


10.5% 


Total 




329 


136 


41.3% 



Three journals accounted for almost half of the source articles (see Table 4) . One fourth of the 
source studies had been published in Public Opinion Quarterly. The Journal of Marketing Research and the 
Journal of Applied Psychology accounted for another 22.5 percent of the studies. Fifty-four journals included 
had published no more than two of the cited source studies. 
Bias Favoring the Journal in Which the Review was Published 

In examining the number of source studies drawn from the publication in which the review articles 
appeared, the findings were somewhat reflective of the overall percentages of articles in the total list of 
329 citations for several of the reviews. Linsky drew 15 (26 percent) of his source articles from Public 
Opinion (Quarterly, the journal in which his review appeared (see Appendix B). Yammarino, Skinner, and 
Childers (1991) and Fox, Crask, and Kim (1988), also published in Public Opinion Quarterly, listed 19 
percent ai:d 18 percent, respectively, of their citations from that journal It must be noted, however, that 25 
percent of the articles in the total list of source articles appeared in Public Opinion Quarterly. 

Kanuk and Berenson (1975) had only 9 percent of their references from the Journal of Marketing 
Research, which accounted for 12,5 percent of the total source arHcle citations. Harvey (1987) and Conant, 
Smart, and Walker (1990), both published in the Journal of the Market Research Society, had 5 percent 
and 4 percent, respectively, of their source articles from that journal which accounted for 3 percent of the 
overall list of source articles. 
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Table 4 

Publication Sources of Studies Cited in Reviews 



Journal 


Number 


Per Cent 


i uDiic \jpinion v»^aneriy 


83 


25.2 


Journal of Marketing Research 


41 


12.5 


Trill •'nol r\( At^t^IicvI T'cv/^Vmlnort/ 

juuiTiai or rvppiicu 1 Dyuiuiugy 


33 


10.0 


journal or iviarKenng 


14 


4.3 


Journal of Advertising Research 


14 


4.3 


/VuicnCan DOClOIUglCal IxcVlcW 


13 


4.0 


Journal of the Market Research Society 


10 


3.0 


Journal or tne Academy or MarKeung Dcience 


7 
# 


2.1 


journal or tne American jtansncai /\i>»uv.iduun 


7 


2.1 


Psychological Reports 


6 


1.8 


DOCioiog)^ ana social ivebearLn 


5 


1.5 


JUurilcU Ui DUdlllcdd I\C9Cc«iL.ii 


4 


1.2 


Journal or 1 ersonality and bociai i sycnoiogy 


4 


1.2 


Printers Ink 


3 


0.9 


American ^4arketing Association Educators' Conference Proceedings 


3 


0.9 


Journals publishing two source articles (10 journals) 


20 


6.1 


Journals publishing one source article (44 journals) 


44 


13.4 


Books 


8 


2.4 


Unpublished iTunuscripts 


5 


1.5 


Studies cited in previous publication but not published independently 


5 


1.5 



Heberlein and Baumgartner (1978) leaned more heavily on the American Sociological Review, in 
which they were published, including 11 percent of their articles from that journal that accounted for only 
4 percent of the total list of citations. 

Two reviews appeared in journals that do not typically publish survey research method studies. 
Bruvold and Comer (1988) had only one of their 212 references from the Journal of Business Research, but 
that journal contributed only four citations to the total list. Duncan (1979), as well as all other review 
authors, included no citations from the Journal of Management, in which his review appeared. 



8 10 



Analytic Procedures 

Heberlein and Baunigartner (1978) based their analysis on procedures introduced by Sudman and 
Bradbum (1974) and a sinrular procedure by Glass and Smith (1976), one of the early works on meta- 
analysis, Bruvold and Comer (1988) endeavored to develop a model to estimate mail survey response rates 
using a weighted-least squares logit regression procedure based on Berkson (1944), Flath and Leonard 
(1979), and Gn^^n, Carmone, and Wachspress (1977). Fox, Crask and Kim (1988) labeled their analysis as 
meta-analysis and described it as similar to that used by Armstrong and Lusk (1987) and Yu and Cooper 
(1983), Yammarino, Skinner and Childers (1991) also used meta-analytic proceduies, citing Glass (1977), 
Hunter and Schmidt (1990), and Hunter, Schmidt, and Jackson (1982) as methodological references. 
Presentation of Findings in Qualitative Reviews 

Linsky (1975), Kanuk and Berenson (1975) and Duncan (1979) utilized tables in summarizing the 
impact of specific manipulations. Conant, Smart, and WalKer (1990) presented a single table summarizing 
the effects of the various techniques. Harvey (1987) used only text descriptions None of the three articles 
appearing in the marketing research journals used quantitative procedures. 

Discussion 

There is considerable variation among the nine review articles broadly examimng mail survey 
response rate facilitation techniques. These are nine independent, sometimes overlapping, reviews rather 
than one review replicated eight times. Procedures are better docunvented in the quantitative reviews, 
although it is recognized that writing style must be geared to the style preferred by the journal to which a 
manuscript is submitted. The existing level of statistical sophistication at the time at which a study is 
conducted must also be taken into account. More recent authors have benefited from the computer search 
capabilities available to them in locating source articles as well as in expanded statistical alternatives. 

The list of 329 source studies cited in these nine reviews is not exhaustive. Each review included one 
or more sltidies not included in others covering the same time span. Other studies of nvH survey methods do 
exist in addition to those cited in these reviews. Those authors who documented their search procedures 
each went beyond previous lists or citation lists in other reviews in their efforts to uncover additional 
research studies. The most thorough searches included computer searches, citation lists in identified 
articles, and manual searches of journals. 

While computer searches facilitate the location of articles, those predating the computer abstract 
databases are less easily found. Potter, Sharpe, Hendee, and Clark (1972), Berdie and Anderson (1974), 
and Pressley (1976) pioneered the compilation of annotated indexes in mail survey research studies and 
each was cited in at least one of the reviews. Indexes to Survey Methodology Literature (U. S. Bureau of 
the Census, 1974) might be helpful in locating early references. The Bibliography of Marketing Research 
Methods^ (Dickinson, 1986) could be a valuable resource in identifying current as well as older articles. 
Dillman and Sangster (1990) cover the years 1974-1989 and provide an update for the extensive 
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bibliography found in Dillman (1978). Public Opinion Quarterly maintains a cumulative index by topic to 
articles it publishes. 

Tl^e choice of keywords or descriptors used in computerized searches of abstract databases 
influences the results. Some of the reviewers may not have discovered relevant articies because the 
terminology used in their searches was too limited. Searches using "response rate" did not produce 
identical lists of citations to those when "response rates'^ was used, except in the ABI/Inform database 
which shortened the plural terms to the singular forms. 

Differences in databases used and their contents also influence the outcomes of the search efforts. 
PsycINFO is an online database of psychological publications that includes dissertations, which are not 
included in PSYCLit. The Social SciSearch database, used by some of the review authors, only permits 
searches of the titles. Social SciSearch is the online version of Social Sciences Citation Index, which is 
very inclusive. Having any type of computerized search process, though limited to title, is a distinct 
improvement over using the bound volumes of the Index. The databases that were searched in the present 
study (ERIC, ABI/Inform, PSYCLit, and Sociofile) were all available on CE)-ROM and all contained 
abstracts as well as titles and descriptors or keywords. In searching the databases, the entire entry 
(including abstract) was searched for the search term(s). 

Fox, Crask, and Kim (1988) expressed concern that usu\^ only published studies might bias the 
results, because studies that failed to find significant effects might not appear in print. They did note, 
however, that published studies were easier to. obtain. Because many studies liave more than one variable 
being manipulated at a time, nonsignificant results for individual factors were more prevalent than 
significant ones in their data. Only 23 percent of the 214 effect estimates in their study were statistically 
significant. 

It would be expected that more recent source studies would appear in only the most recent review. 
Differing selection criteria and search procedures may account for the high incidence of source studies dted 
in only a single review, rather than the apparent failure to capitalize on previous reviews. 

Few journals have been consistent publishers of mail survey response rate studies through the years 
(see Appendix A). The number of journal articles found by the reviewers may, to some degree, be related to 
the search procedures employed by the reviewers. Those journals searched most thoroughly (manually, 
issue by issue) had the largest numbers of citations in the overall list of 329 source citations. The ten 
journals manually searched by Yammarino, Skinner, and Childers (1991), are the ten journals with the 
largest numbers of citations in the total list of source articles. Similarly, the three journals searched by 
Fox, Crask, and Kim (1988) were the three with the largest numbers of source article citations, and four of 
the five journals thoroughly searched by Bruvold and Comer (1988) are the most frequently cited journals in 
the total list. 

The extent to which reviewers used articles published in the journals in which their reviews 
appeared was generally not inconsistent with the proportion of source citations from that journal in the 
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overall list. The publication date of the review and the number of appropriate articles previously 
published in the same journal are subject to considerable variation, making it difficult to draw conclusions 
regarding the extent to which use of references from the review journal have an impact on acceptance for 
publication. The percentage of source articles from the respective review journals, however, does not 
greatly exceed the total percentage of overall source articles appearing in the specific review journals. 

Publication of studies on mail surveys have been published primarily in journals with audiences in 
public opinion research and marketing. Sociological and psychological journals have also been open to such 
studies, but to a lesser extent. By broadly defining eJucstional journals (Journal of Educational Research, 
Educational Research Bulletin, Journal of Educational Psychology, Research in Higher Education, 
Vocational Guidance Quarleriy, Journal of Experimental Education, and College Student Journal), there was 
a total of 10 articles (3 percent) in the list of 329 source citations that can be attributed to any educational 
journals. It is small wonder that education was omitted from Dillnrmn's (1991) statement that "Statistics, 
psychology, marketing rese^-^^^,, economics, and the various health sciences are disciplines in which 
research efforts to improve mail survey methods are regularly conducted and reported" (p. 226). 

It is discouraging to note that no review articles were located in education journals through the 
ERIC search. Another observation that can be made at the conclusion of tiiis study is tiiat tiiere was not a 
single article on the source citation list of 329 that was published in an American Educational Research 
Association (AERA) journal (i.e., American Educational Research Journal, Review of Educational Research, 
Journal of Educational Statistics, Educational Evaluation and Policy Analysis, Review of Research in 
Education). This is particularly disturbing to those who consider AERA as the foremost organization for 
researchers in the field of education. While AERA publishes the Journal of Educational Statistics, this 
publication does not provide a forum for studies on design issues relating to research on mail survey 
methods. Included in the 137 ERIC citations resulting from the computerized search were 20 papers on 
survey research meti^todology that had been presented at annual meetings of AERA. This finding supports 
the contention that the descriptors used in the search were relevant. 

Much can be learned from the review articles. This study calls attention to the disparity with 
which the procedures used in the integrated review articles were documented. The next step in this study 
will be to examine and compare the mail survey response rate variables that were studied in the reviews 
and the findings of the reviewers regarding those variables. 
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Appendix A 

Journal Publication of Source Articles Aaoss Time 



PiihliratinnVfar 



Journal 



1921 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 
- 39 - 45 - 49 - 54 - 59 - 64 - 69 - 74 - 79 - 84 - 89 



total* 



Journal of Applied Psychology 4 
Journal of Juvenile Research 1 
Journal of Educational Psychology 2 
Joiimal of the Am. Statistical Assn. 1 
Pedagogical Seminary 1 
Printers Ink 
Psychological Bulletin 
American Sociological Review 
Journal of Educational Research 
Educational Research Bulletin 
Journal of Psychology 
Public Opinion Quarterly 
Public Opinion Quarterly citation 
Advertising Agency 
Sociology and Social Research 
Journal of Marketing 
Journal of Abnormal/Sodal Psychology 
Proceedings of the Iowa Academy of Science 
boolcs 

Applied Statistics 

Journal of Advertising Research 

American Journal of Sociology 

Journal of Marketing Research 

Journal of the Royal Statistical Society 

Bulletin of the British Psychological Society 



1 

8 



4 2 



1 

1 
2 
1 
1 
3 



♦Includes one undated, unpublished paper 

Note. Ten most frequently dted journals are in bold print. 



2 
1 



1 10 8 



3 3 12 1 

1 



7 12 8 15 18 8 
1 1 3 



1 2 1 

4 1 5 2 



1 2 

2 7 13 1 
1 

1 5 3 20 12 



33 
1 
2 
7 
1 
3 
1 

13 
2 
1 
1 

83 
5 
1 
5 

14 
1 
1 
8 
1 

14 
1 

41 
1 
1 
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Appendix A. Journal Fublicaiion of Source Articles Across Time (continued) 



PiihliratinnVpar 



1921 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 
Journal - 39 - 45 - 49 - 54 - 59 - 64 - 69 - 74 « 79 - 84 - 89 total* 



Journal of Consulting Psychology 1 1 

Am. Statistical Assn. Proceedings 1 1 

Industrial Marketing 112 

Am. Journal of Public Health 1 1 

Rural Sociology 1 1 

British Journal of Psychiatry 1 1 

British Journal of Preventive & Social Medidne 1 1 

Journal of Personality and Social Psychology 13 4 

Journal of Counseling Psychology 1 1 

Social Forces 1 1 

U.S. Govt Printing Office 1 1 

Journal of the Market Research Society 3 12 4 10 

Journal of Business Communication 11 2 

Akron Business & Economic Review 1 1 

American Psychologist 1 1 

Journal of Forestry 1 1 

Vocational Guidance Quarterly 1 1 

Psychonomic Science 1 1 

Research in Higher Education 11 2 

Social Science Quarterly 2 2 

Pacific Sociological Review 1 1 

Psychological Reports 112 2 6 

Journal of Experimental Social Psychology 1 1 

Ag. Dept., U. of Wisconsin 1 1 

Journal of Business Research ^ 14 

Joumalof the Academy of Marketing Science 2 2 3 7 



*Includes one undated, unpublished paper 

Note. Ten most frequently cited journals are in bold print. 
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Appendix A. Journal Publication of Source Articles Aaoss Time (continued) 



Journal 



1921 1940 1945 1950 1955 1960 1965 1970 1975 1980 1985 
- 39 - 45 - 49 - 54 - 59 - 64 - 69 - 74 - 79 - 84 - 89 total* 



Sociological Quarterly 

Arkansas Business & Economic Review 

Management Science 

Personnel Psychology 

Journal of Business 

Journal of Consumer Research 

Revista Espanola de la Opinion Publica 

Canadian Review of Sociology and Anthropology 

Sociological Methods and Research 



2 



2 



1 



1 



1 



1 



1 



1 



1 



Journal of Advertising 

Southern Marketing Assn. Proceedings 

European Research 

College Student Journal 

unpublished 

Proceedings of the Am. Mktg. Assn. 
Perceptual and Motor Skills 
Zeitschrift fur Soziologie 
Industrial Marketing Management 
Journal of Exerimental Education 
Quarterly Review of Marketing 
Journal of Marketing Education 
unpublished and undated 

Totals 9 10 12 21 21 25 32 49 83 

♦Includes one undated, unpublished paper 

Note, Ten most frequently cited journals are in bold print. 



1 
2 
1 
1 

2 2 
3 
1 
1 

1 1 

1 

1 



50 17 329* 
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Appendix B 
Soxirce Article Pub^cation Sources 



RgykwArtide 



Journal Lin Kan HebDunc Harv Bruv Fox Con Yam Total 



AMA Educators' Conference Proceedings 












1 


1 


1 


3 


ASA Proceedings 








1 










1 


Advertising Agency 


1 






1 










1 


Ag. Dept., U. of Wisconsin 
















1 


1 


Akron Business and Economic Review 










1 








1 


American Journal of Public Health 






1 












1 


American Journal of Sociology 




1 


1 












1 


American Psychol gist 










1 








1 


American Sociological Review 


5 


5 


10 


10 


12 






4 


13 


Appiiea statistics 








1 










1 


Arkansas Business and Economic Review 








1 










1 


British Journal of Preventive & Social Medicine 








1 










1 


British Journal of Psychiatry 








1 












DUiieun or tne Dntisn i sycnoiogicai bociety 


1 






1 


i 
1 








I 


Canadian Review of Sociology & Anthropology 














1 




1 


College Student Journal 








1 










1 


Educational Research Bulletin 


1 




1 


1 


1 








1 


European Research 








1 




1 


1 




1 


Industrial Marketing 


1 






1 










2 


Industrial Marketing Management 














2 




2 


Journal of Abnormal Social Psychology 
















1 


1 


Journal of Advertising 








1 






1 




1 


Journal of Advertising Research 


5 


10 


4 


5 5 


9 


5 


2 


9 


14 


Journal of Applied Psychology 


6 


6 


14 


7 11 


27 


2 


3 


11 


33 


Journal of Business 










1 




1 


1 


1 



Note: Review articles abbreviated as follows: Linsky, 1975 (Lin); Kanuk & Berenson, 1975 (Kan); 
Heberlein & Baumgartmer, 1978 (Heb); Duncan, 1979 (Dune); Harvey, 1987 (Harv); Bruvold & Comer, 1988 
(Bruv); Fox, Crask, & Kim, 1988 (Fox); Conant, Smart, & Walker, 1990 (Con); Yammarino, Skinner, & 
Childers, 1991 (Yam), 
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Appendix B. Source Article Publication Sources (continued) 



Journal Lin Kan HebDunc Harv Bniv Fox Con Yam Total 



JOiUTlAi Ur OUdiricao v«UllullUlllL.clliUIi 








1 


2 


1 


1 




1 


2 


journal or DUdincSd jxtrdi^arcn 








1 


1 


1 


1 


3 


2 


4 


joumcii Oi V-Onsuiung i sycnoiogy 




1 
1 
















1 


journal or L^onsurner jvesearcn 










1 

Jl 


1 








2 


Journal of Counseling Psychology 


1 








1 

£ 










1 


journal or caucationai i sycnoiogy 






1 

1 






1 






1 


2 


journal or caucanonai ixcbcarcn 


1 

A 




2 






2 






1 


2 


journal or nxpenmeniai nuucanon 




















1 


journal or nxpcnnicniai social 






















Psychology 












1 








1 
1 


Journal of Forestry 






1 






1 








1 


Journal of Juvenile Research 






1 






1 








1 


Journal of Marketing 


5 


6 


6 


2 


9 


13 


3 


4 


8 


14 


Joi: mal of Marketing Education 
















1 




1 


Journal of Marketing Research 


2 


6 


2 


15 


12 


38 


15 


19 


17 


41 


Journal of Personality and Sodal Psychology 












4 








4 


Journal of Psychology 




1 


1 














1 


Journal of the Academy of Marketing Science 










1 


2 


2 


4 


4 


7 


Journal of the American Statistical Assn. 


3 


1 


1 


1 


3 


4 






3 


7 


Joumal of the Market Research Society 










6 


4 


1 


2 


8 


10 


Journal of the Royal Statistical Society 


1 


1 


1 


1 


1 


1 








1 


Management Science 


















1 


1 


Pacific Sociological Review 






1 






1 








1 


Pedagogical Seminary 






1 






I 








1 


Perceptual and Motor Skills 










1 










1 


Personnel Psychology 


















1 


1 


Proceedings of the Iowa Academy of Science 






1 






I 








1 



Note: Review articles abbreviated as follows: Linsky, 1975 (Lin); Kanuk & Berenson, 1975 (Kan); 
Heberlein & Baumgartmer, 1978 (Heb); Duncan, 1979 (Dune); Harvey, 1987 (Harv); Bruvold & Comer, 1988 
(Bruv); Fox, Crask, & Kim, 1988 (Fox); Conant, Smart, & Walker, 1990 (Con); Yammarino, Skinner, & 
Childers, 1991 (Yam). 
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Appendix B. Source Article Publication Sources (continued) 



RpvipwAttidfi 



ERIC 



Journal Lin Kan Heb DuncHarv Bruv Fox Con Yam Total 



1 uDlic ijpinion Vijuaneriy 


IS 2Q 38 20 

1%J 4b7 fc/U 


31 


66 


7 7 22 


83 


Public Opinion Quarterly citation only 








5 


5 


Printers Ink 






1 

A 




3 


Psychological Bulletin 


1 








1 


Psychological Reports 


1 

A 




1 


3 


6 


1 * - * - 

Psychonomic bcience 






\ 


1 


1 


Quarterly Review of Marketing 




1 
A 






i 


Research in Higher Education 




1 
A 




1 


2 


Revista Espanola de la Opinion 


1 
i 








1 


Rural Sociology 


1 

X 




1 
A 




1 


Social Forces 


1 11 
X XI 


1 


1 
A 


1 


1 


Social Science Quarterly 


1 




2 


1 


2 


Sociological Methods & Research 


2 








2 


Sociological Quarterly 




1 






1 


Sociology and Social Research 


2 4 1 


3 


5 




5 


Southern Marketing Assn. Proceedings 








2 


2 


U.S. Govt. Printing Office 




1 






1 


Vocational Guidance Quarterly 


1 1 


1 


1 


1 1 


1 


Zeitschrift fur Soziologie 








1 


1 


books 


2 1 


5 






8 


nnanuscripts (unpublished) 


1 


3 




2 


5 


Note: Review articles abbreviated as follows: Linsky, 1975 (Lin); Kanuk & Berenson, 1975 (Kan); 



Heberlein & Baumgartmer, 1978 (Heb); Duncan, 1979 (Dune); Harvey, 1987 (Harv); Bruvold & Comer, 1988 
(Bruv); Fox, Qask, & Kim, 198^ (Fox); Conant, Smart, & Walker, 1990 (Con); Yammarino, Skinner, & 
Childers, 1991 (Yam). 
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